The present invention relates to methods of correcting distortions generated in a taken image as distortions, which are caused by a hand movement when the image is taken by using an image-taking apparatus, and relates to apparatus for correcting such distortions by adoption of the methods. The present invention can be well applied to an image-taking apparatus such as a video camera or a digital still camera and a recording/reproduction apparatus having an image-taking function.
When a moving picture is taken by using a carried video camera, for example, a positional displacement of an image-taking device employed in the video camera may occur in the horizontal and/or vertical directions of the taken image due to a so-called hand movement. In this case, the positional displacement appears on the taken image as an image wabble.
That is to say, if there is no hand movement, a position at which a taken image is formed does not change on an image-taking plane. In other words, if a plurality of frame images is arranged in the direction of the time axis, the frame images coincide with each other completely at the same position as shown in FIG. 21A. If there is a hand movement, on the other hand, a position at which a taken image is formed changes on an image-taking plane. In other words, if a plurality of frame images is arranged in the direction of the time axis, the frame images form swaying images shown in FIG. 21B.
The phenomenon of this hand movement occurs strikingly with ease when a zoom lens is used on the telescope side. When the phenomenon of this hand movement occurs, a standstill thing appears to swing, raising a problem of a difficulty to view the image. In addition, the swinging appearance of the standstill thing also causes a phenomenon of an image showing a blurred object of photographing.
In the case of the conventional electronic image-taking apparatus, data read out from an image-taking device employed in the apparatus as data representing a taken image is subjected to digital processing carried out as a process to correct distortion of the image. By carrying out such digital processing, it is possible to correct distortion caused by a hand movement as distortion of the image.
In the conventional image-taking apparatus such as a video camera or a digital still camera and the conventional recording/reproduction apparatus having an image-taking function (or the conventional cell-phone having a camera or the conventional personal computer having a camera), as the image-taking device, a solid-state image-taking device employing a CCD (Charge Couple Device) is widely used in general. In the following description, the solid-state image-taking device employing a CCD is referred to as a CCD imager.
As described in documents such as Japanese Patent No. 3384459, which is used as patent document 1 in this specification, a large number of technologies for correcting distortions caused by a hand movement on the assumption that a CCD imager is used as an image-taking device has been proposed. As a matter of fact, products each adopting a technology for correcting distortions caused by a hand movement are widely sold in world markets. The technology for correcting distortions caused by a hand movement is referred to hereafter simply as a hand-movement correction technology.
The conventional hand-movement correction technology makes use of a characteristic that a time to sample the quantity of light accumulated in all pixels of the CCD imager is uniform throughout the structure of the CCD imager. That is to say, a sampling operation is carried out once per frame.
To put it in detail, in the CCD imager, all pixels are exposed at the same time and the whole image data of one frame is fetched with just a single timing. Thus, for all pixels of one frame, only one hand-movement displacement Vcs shown by an arrow in FIG. 22 can be considered. That is to say, a taken-image distortion caused by a hand movement as a distortion of a frame of a taken image can be corrected by correction of a reading pixel position (or a sampling pixel position) by the hand-movement displacement Vcs, which can be detected for the frame as a distortion caused by a displacement from an area Fla enclosed by a solid line in FIG. 22 as the accumulation area of the original frame of the taken image to an area FLb enclosed by a dash line in the image due to the hand movement.
It is to be noted that, in general, not all pixels of an image-taking device are treated as effective pixels as shown in FIG. 22. Instead, only pixels in a central part of the entire area AFL are used as a valid image area EFL. The entire area AFL is referred to as an effective image area including all pixels. Defined by a vertical valid length and a horizontal valid length, the valid image area EFL is obtained by excluding peripheral areas from the effective image area AFL.
With such an imager used, by using only data of pixels originally owned by the imager, it is possible to correct a distortion from the taken image as a distortion, which is resulted in when the reading pixel position changes due to a hand movement, provided that the magnitude of the hand movement is within a range smaller than the difference between the effective image area AFL and the valid image area EFL. Thus, in comparison with interpolation processing or the like in which data necessary for correction of a distortion caused by a hand movement is generated, the amount of image deterioration decreases.
By the way, in recent years, as the image-taking device of an electronic image-taking apparatus, a solid-state image-taking device having an X-Y address type is employed. The solid-state image-taking device having an X-Y address type is an image-taking device capable of outputting data of any pixel unit on the screen by specifying the horizontal-direction and vertical-direction positions of the pixel. The horizontal-direction position is also referred to as an X-direction position and the vertical-direction position is also referred to as a Y-direction position. An example of the solid-state image-taking device having an X-Y address type is a CMOS-type solid-state image-taking device, which is referred to hereafter as a CMOS imager.
The CMOS imager has the following features:
(a) The CMOS imager is an imager of an amplifying type. Since an amplified signal is read out, the imager offers a high sensitivity
(b) Since a CMOS circuit is employed, the power consumption is low
(c) The CMOS imager can be manufactured at a low cost
(d) As a rule, the CMOS imager is capable of outputting data of any pixel selected at random or making an access to any pixel selected at random.
Thus, even though the CMOS imager is capable of outputting data of a taken image in pixel units, in practicality, the CMOS imager generally outputs (or samples) the data of a taken image in pixel-group units each including pixels on one horizontal line in a process to output the data.
If the CMOS imager outputs data of a taken image in horizontal-line units as described above, as shown in FIG. 23, there is a time shift Δt between exposure times of adjacent horizontal lines. The time shift Δt is equal to a difference between in reading time between the adjacent horizontal lines. It is to be noted that, if the CMOS imager outputs data of a taken image in pixel units, a difference in reading time between adjacent pixels is so smaller than the difference between in reading time between adjacent horizontal lines so that the difference between in reading time between adjacent pixels can be ignored. Even if the CMOS imager outputs data of a taken image in pixel units, however, there is still a similar time shift between exposure times.
Because of what is described above, in the case of an image-taking apparatus employing a CMOS imager, for an image that should naturally be obtained as shown in FIG. 24A as an image showing a vertically erected house and a vertically erected tree, when the image is taken from a position inside a running train, for example, an image of a slanting house and an inclined tree is actually obtained as shown in FIG. 24B. The difference between the images shown in FIGS. 24A and 24B is a result of the so-called focal plane phenomenon peculiar to the CMOS image.
The typical image shown in FIG. 24 (B) is an image, which is taken while the image-taking apparatus is moving in the horizontal direction. If the image of a photographing object is taken while the image-taking apparatus is moving in the vertical direction, however, the resulting image will show the object shrunk or expanded in the vertical direction as shown in none of the figures.
The phenomenon described above occurs strikingly when the relative velocity between the object of photographing and the image-taking apparatus employing the CMOS imager. For example, the phenomenon described above occurs strikingly when the photographer takes an image of the photographing object by using the image-taking apparatus while moving at a high velocity or, conversely, when the photographer staying at a fixed location takes an image of the photographing object, which is moving at a high velocity. In the case of ordinary photographing, however, such a situation can be said to be a situation that occurs rarely.
When the photographer takes an image of the photographing object by using the image-taking apparatus held in its hand, however, the hand may move a little bit at a high velocity. In this case, the focal plane phenomenon described above occurs due to the hand movement.
This is because the value of a hand movement happening to a CMOS imager is not one value in a frame as is the case with a CCD imager but, as described above, a value varying from pixel to pixel or horizontal line to horizontal line due to differences in sampling time between the pixels in the frame or between the horizontal lines in the frame. For this reason, in an image-taking apparatus employing a CMOS imager, a distortion caused by the focal plane phenomenon described above remains uncorrected even if a correction process merely using a hand-movement quantity for one frame is carried out. In the following description, a hand movement happening to a CMOS imager is also referred to as a CMOS hand movement.
In the case of an image-taking apparatus employing a CMOS imager, a queer squashy distortion like one shown in FIG. 24C is observed in the output of a taken image of a photographing object in the event of a hand movement due to the fact that the direction, magnitude and velocity of the hand movement are not uniform in a frame or the taken image.
By the way, in the case of an apparatus for taking a still image, as preconditions for still-image photographing, it is assumed that the magnitude of a hand movement is limited and a mechanical shutter is employed so that a focal plane phenomenon caused by a CMOS hand movement can be avoided relatively with ease. An example of the apparatus for taking a still image is a digital still camera.
In the case of another kind of image-taking apparatus such as a video camera, on the other hand, a moving picture photographing is assumed. Thus, a moving picture-taking apparatus for business applications or for a moving picture-taking apparatus of a high-class category adopts a method for virtually preventing a focal plane phenomenon from occurring due to a CMOS hand movement. In accordance with this method, a maximum difference in sampling time in one frame is reduced and the data of a taken image is read out at a very high velocity.
In addition, the magnitude of a hand movement relative to an object of photographing rises as the magnification of an optical zoom is increased. Thus, if the image-taking apparatus does not have an optical-zoom function or the image-taking apparatus has an optical-zoom function but the magnification of the optical zoom is small, the CMOS hand movement does not raise a big problem even in an application of taking moving pictures. In the first place, for an inexpensive image-taking apparatus where even a distortion caused by a hand movement is not corrected by using an acceleration sensor as is the case with the conventional image-taking apparatus employing the CCD imager, a bad effect caused by the CMOS hand movement is relatively small, not even raising a problem in many cases.
In order to solve the problems described above, there has been demanded a technology for correcting and avoiding a distortion of a focal plane phenomenon caused by a CMOS hand movement as a technology that can be implemented with a high degree of precision at a low cost in an image-taking apparatus having a high-magnification optical zoom function for taking moving pictures as a main application by carrying out digital-signal processing and with neither employing a special device nor utilizing a high-velocity sampling clock signal.
By the way, Japanese Patent Laid-open No. 2004-266322 serving as patent document 2 describes a method to correct a distortion including a focal plain phenomenon caused by a CMOS hand movement in a taken image.
In accordance with the distortion correction method described in patent document 2, when data of a taken image is read out from the CMOS imager in horizontal-line units, a positional displacement quantity caused by a generated hand movement for each horizontal line is detected and a correction is made so as to read out the data of the taken image from a position shifted from the original position by the detected positional displacement quantity in a direction opposite to the direction of the hand movement. The positional displacement quantity is also referred to hereafter as a hand-movement quantity.
However, this method raises a problem of a difficulty to obtain a hand-movement quantity for each horizontal line as a problem caused by conditions such as the sampling frequency of a sensor for detecting a hand movement. In order to solve this problem, in accordance with the distortion correction method described in patent document 2, a hand-movement quantity is detected discretely in the vertical direction of the screen every plurality of horizontal lines as shown in FIG. 25A. In the example shown in FIGS. 25A to 25C, hand-movement quantities Q1, Q2, Q3 and so on are detected every 50 horizontal lines as shown in FIG. 25B. It is to be noted that the figure shows only hand-movement quantities in the horizontal direction.
The hand-movement quantities of the remaining 49 horizontal lines not directly subjected to the process to detect hand-movement quantities, that is, the hand-movement quantities of the horizontal lines other than the horizontal line subjected to the process to detect hand-movement quantities, are found by interpolation based on the hand-movement quantities Q1, Q2, Q3 and so on. As the interpolation method, it is possible to adopt some techniques, one of which is shown in FIG. 25C. In accordance with the interpolation method shown in FIG. 25C, basically, the interpolation to find a hand-movement quantity of any specific one of 49 horizontal lines not subjected to the process to detect hand-movement quantities is based on 2 hand-movement quantities Qn and Qn+1 where notation n in the subscript denotes an integer at least equal to 1. The hand-movement quantity Qn is the hand-movement quantity of a horizontal line immediately preceding the 49 horizontal lines while the hand-movement quantity Qn+1 is the hand-movement quantity of the a horizontal line immediately succeeding the 49 horizontal lines.
In accordance with interpolation method (1) shown in FIG. 25C, for example, the detected hand-movement quantity Qn of the horizontal line immediately preceding the 49 horizontal lines is used as it is for the horizontal in the first half of the 49 horizontal lines. On the other hand, the detected hand-movement quantity Qn+1 of the horizontal line immediately succeeding the 49 horizontal lines is used as it is for the horizontal in the second half of the 49 horizontal lines.
In accordance with interpolation method (2) shown in FIG. 25C, on the other hand, values represented by a straight line connecting the detected hand-movement quantity Qn of the horizontal line immediately preceding the 49 horizontal lines and the detected hand-movement quantity Qn+1 of the horizontal line immediately succeeding the 49 horizontal lines are used as hand-movement quantities of the 49 horizontal lines. That is to say, interpolation method (2) is the so-called average value interpolation method.
By adoption of any of the distortion correction methods described in patent document 2, it is possible to correct a distortion including a focal plain phenomenon caused by a CMOS hand movement in an image taken by using a CMOS imager.
However, even though patent document 2 describes a method to reduce a focal plane phenomenon on one screen (or one frame) of a taken image, the reference does not disclose a method to eliminate an effect of an inter-frame hand movement described above in the case of a moving picture. Thus, with only the disclosed method, it is impossible to eliminate the effect of an inter-frame hand movement described above in the case of a moving picture.
If the technology disclosed in patent document 1 is adopted, on the other hand, an inter-frame hand movement can be corrected. However, it is impossible to correct an image distortion such as the focal plane described above in a frame.